22 Nominal Predicted Variable

22.1 Softmax Regression

• Two usual types of generalization:
  • Conditional logistic regression (later in the chapter)
  • Multinomial logistic regression (this section). We refer to this method as softmax regression because logistic is a misnomer as doesn’t use the logistic function per se, also all the models in this chapter describe multinomial data so multinomial isn’t informative.

Suppose we have a single metric predictor \(x\). Underlying linear propensity of outcome k is denoted \(\lambda_k = \beta_{0,k} + \beta_{1,k}x\). The probability of outcome k is then given by: \[\phi_k = softmax_S(\{\lambda_k\}) = \frac{exp(\lambda_k)}{\sum_{c\in S}exp(\lambda_c)}\] Why exponentiate? Authors explanation: need to have non-negative and preserve order. BB: this is not sufficient explanation! How can we really call this a probability when there is not more theoretical backing because surely this greatly distorts how how larger larger propensities seem compared to smaller ones. Important to note that for any logistic regression we are just making a model of probabilities and these models aren’t really necessarily very accurate; the accuracy of probability estimates is often actually far worse than the accuracy of the class predictions.

\[\text{As } \gamma \rightarrow \infty, \frac{exp(\gamma\lambda_k)}{\sum_{c\in S}exp(\gamma\lambda_c)} \rightarrow \begin{cases} 1 & \text{if } \lambda_k = max(\{\lambda_c\}) \\ 0 & \text{otherwise} \end{cases} \]

With very high gamma, softmax just assigns maximum input. Useful for applications that use the derivative to optimize as it is smooth and differentiable.

There are indeterminancies in the system of equations: we can add a constant \(C_0\) to every \(\beta_{0,k}\) and \(C_1\) to every \(\beta_{1,k}\) and get the same probabilities. Therefore we can set the baseline and slope of one of the categories to arbitrary convenient constants. We will set the constants of one response category, called the reference category \(r\), to zero.

Because of the indeterminacy of the regression coefficents we can interpret the regression coefficientso nly relative to the reference category. The regression coefficients can be conceived in terms of the log odds of each outcome relative to the reference outcome. \(log(\frac{\phi_k}{\phi_r})=\beta_{0,k}+\beta_{1,k}x\)

22.2 Conditional logistic regression

We can divide the set of outcomes into a hierarchy of two-set divisions, then use a logistic to describe the probability of each branch of the two-set divisions. See figure 22.2 for best demonstration.

Remember that each \(\phi\) is a conditional probability so we get structures such as: \(\phi_2 = \phi_{[2]|[2,3,4]} (1-\phi_{[1]|[1,2,3,4]})\).

Results in boundaries that definitely demarcate based on the order of divisions chosen to make up the hierarchy.

In general, conditional logistic regression requires that there is a linear division between two subsets of the outcomes, and then within each of those subsets there is a linear division of smaller subsets, and so on. This sort of division is not required of the softmax regression model.

With many predictors can be virtually impossible to visually ascertain which sort of model is most appropriate; choice of model is driven primarily by theoretical meaningfulness.

22.3 Implementation in JAGS

22.4 Generalizations and Variations of the Models

22.5 Exercises

setwd("./DBDA2Eprograms")
source("DBDA2E-utilities.R") # Load definitions of graphics functions etc.
source("Jags-Ynom-XmetMulti-McondLogistic1-Example.R")